Providing a model of impact of a cooling infrastructure

ABSTRACT

A model is provided that produces predicted sensor data as a function of at least one input feature that includes an adjustable setting of a cooling infrastructure. The model is able to model a non-linear relationship between the predicted sensor data and the adjustable setting.

This application is a PCT national stage application under 35 U.S.C. §371 of PCT Application No. PCT/US2012/039694, filed May 25, 2012.

BACKGROUND

A data center can include an arrangement of electronic devices,including processing servers, storage servers, communication nodes, andso forth. The electronic devices can be arranged in racks provided in aroom (or multiple rooms). To provide temperature control, coolingdevices, such as computer room air conditioning (CRAC) units can beprovided to manage the cooling of the electronic devices. Adjustablesettings associated with the cooling devices can be controlled toprovide cooling at specific locations and to a specific degree asneeded.

BRIEF DESCRIPTION OF THE DRAWINGS

Some embodiments are described with respect to the following figures:

FIG. 1 is a flow diagram of a model producing process according to someimplementations;

FIG. 2 is a schematic diagram of an example data center having racks ofelectronic devices, and a controller according to some implementations;

FIG. 3 is a flow diagram of an analysis process according to someimplementations;

FIG. 4 illustrates a regression tree model according to some examples;

FIG. 5 illustrates an example layout of a data center;

FIG. 6 is a graph illustrating thermal correlation indexes as a functionof temperature sensors, generated according to some implementations; and

FIG. 7 is a block diagram of an example system that incorporates someimplementations.

DETAILED DESCRIPTION

To efficiently manage the adjustable settings associated with a coolinginfrastructure for a data center or any other arrangement of electronicdevices (which can include processing servers, storage servers,communication nodes, and so forth), a relationship between theadjustable settings of the cooling infrastructure and the cooling impactat particular locations (of temperature sensors) is derived. The coolinginfrastructure can include various cooling devices, such as computerroom air conditioning (CRAC) units or other types of cooling devices. Acooling device can have one or multiple adjustable settings. Examples ofadjustable settings of a cooling device include one or any combinationof the following: a temperature setting of the cooling device (whichcontrols the output or supply air temperature of the cooling device), ablower speed (which specifies a speed of a blower of the coolingdevice), and so forth. An example of the blower speed is the blowervariable frequency drive (VFD) speed.

The cooling infrastructure can also include conduits (such as plenumsbelow a raised floor or other types of airflow conduits) to direct coldair from cooling devices to the electronic devices. In some examples,conduits can be associated with one or multiple adjustable settings. Anexample of such an adjustable setting can include an adjustable openingin the tile of a raised floor on which the electronic devices aresupported. Such openings are referred to as vent tile openings.

There can be other adjustable settings associated with the coolinginfrastructure in other examples.

A cooling infrastructure can consume a relatively large amount of energyin a data center. As a result, it is desirable to efficiently manage thecooling infrastructure to provide cooling to the extent needed (in termsof location, time, and amount of cooling). Temperature sensors areprovided at various locations to make temperature measurements. Inresponse to the measured temperatures, the adjustable settings of thecooling infrastructure can be controlled to provide the requisitecooling.

To efficiently manage the cooling infrastructure in response to thetemperature measurements, the relationship between adjustable settingsof a cooling infrastructure and the cooling impact at temperature sensorlocations is derived. In some examples, such relationship is manuallygenerated. Manual generation of such relationship can be atime-consuming and disruptive process, since each cooling device isindependently perturbed with the response at each temperature sensorlocation measured.

In accordance with some implementations, data-centric machine-learningbased techniques or mechanisms are provided for determining arelationship between cooling infrastructure adjustable settings andcooling impact at respective temperature sensor locations. In someimplementations, the relationship can be expressed using thermalcorrelation indexes (TCIs). A TCI is a measure of a relationship betweena change in temperature at a specific temperature sensor (or temperaturesensor location) and a corresponding change in an adjustable setting (orgroup of adjustable settings) of a component of a coolinginfrastructure.

In other implementations, instead of or in addition to determining arelationship between cooling infrastructure adjustable settings andtemperature, a relationship can be determined between coolinginfrastructure adjustable settings and another parameter, such aspressure, relative humidity, and so forth. Thus, more generally, a“correlation index” can be determined that is a measure of arelationship between a change in a parameter (or multiple parameters),such as temperature, pressure, relative humidity, and so forth, and acorresponding change in an adjustable setting (or group of adjustablesettings).

In the ensuing discussion, although reference is made to TCI,temperature sensor data, and temperature sensors, it is noted thattechniques or mechanisms as described are also applicable in the contextof other parameters and use of other types of sensors.

TCI values can be used for identifying, for any given temperature sensorlocation, which of multiple cooling devices have an influence on thegiven temperature sensor location. For example, a first cooling devicethat is relatively close to the given temperature sensor locations wouldlikely have a greater influence on a temperature at the giventemperature sensor location than a second cooling device that is locatedfarther away from the given temperature sensor location. Thus, using TCIvalues, regions of influence for each of the cooling devices can bedetermined. The determined TCI values and regions of influence can beused by a controller for efficiently managing a cooling infrastructurein response to temperature measurements from temperature sensors.

The machine learning-based techniques or mechanisms according to someimplementations involve plural stages. A first of the plural stagesproduces a model for each temperature sensor location, where the modelproduces predicted temperature sensor data as a function of at least oneinput feature that includes an adjustable setting (or multipleadjustable settings) of the cooling infrastructure. A second of theplural stages uses models generated for respective temperature sensors(or temperature sensor locations) for deriving TCI values and regions ofinfluence of cooling devices. Non-intrusive techniques or mechanisms(that do not involve actual perturbations of cooling infrastructureadjustable settings for measuring impact at temperature sensorlocations) are thus provided to determine a relationship between coolinginfrastructure adjustable settings and cooling impact at respectivetemperature sensor locations.

FIG. 1 is a flow diagram of a process 100 of producing a model accordingto some implementations (first stage noted above). The process 100receives (at 102) recorded data (historical data that was previouslyrecorded), where the recorded data includes temperature sensor data(collected by one or multiple temperature sensors) and an adjustablesetting (or multiple adjustable settings) of a cooling infrastructure.Examples of the adjustable settings include a temperature setting of acooling device, a blower speed of the cooling device, a setting of anadjustable opening associated with an airflow conduit, and so forth. Inother examples, the recorded data can instead or additionally includesensor data for another parameter, such as pressure, relative humidity,and so forth.

The process 100 generates (at 104) a model of an impact of the coolinginfrastructure based on the recorded data, where the model producespredicted temperature sensor data as a function of at least one inputfeature that includes an adjustable setting (or adjustable settings) ofthe cooling infrastructure. More generally, the model produces predictedsensor data (temperature, pressure, relative humidity, etc.) as afunction of the at least one input feature.

In some examples, the model does not assume a linear relationshipbetween sensor data and adjustable settings of the coolinginfrastructure. In fact, the model is able to model a non-linearrelationship between predicted sensor data and an adjustable setting. Insome implementations, a separate model can be generated for each sensor(or sensor location).

In further implementations, discussed further below, instead ofgenerating just one model for each sensor location, an ensemble ofmultiple models can be generated for each sensor location.

FIG. 2 is a schematic diagram of an example arrangement, which can bepart of a data center that includes various electronic devices 202. Theelectronic devices 202 are arranged in multiple racks 204, where a rackcan refer to a support structure for holding a respective collection ofelectronic devices 202. The racks 204 can be arranged inside a room 200,which can also include a cooling device 206. Although just one coolingdevice 206 is depicted in FIG. 2, note that there can be additionalcooling devices. In other examples, the cooling device 206 can belocated outside the room 200.

The cooling device 206 has a blower 207 (for generating airflow).Adjustable settings associated with the cooling device 206 include anadjustable blower speed of the blower 207, and an adjustable output orsupply temperature of cold air output from the cooling device 206.

The room 200 has a raised floor 208 on which the racks 204 aresupported. The raised floor 208 allows for a plenum 210 to be providedunderneath the raised floor 208. The plenum 210 can be used forcommunicating cold air from the cooling device 206 to the racks 204. Inother examples, other types of airflow conduits can be used fordirecting cold air from the cooling device 206 to the racks 204.

Note that FIG. 2 depicts one type of data center that has the raisedfloor 208 and that has a generally open environment. There can be otherdesigns for which techniques or mechanisms according to someimplementations are applicable, such as contained environments, pods,non-raised floor environments with cool air delivered from above, and soforth.

The racks 204 can be arranged in multiple rows, such that cold and hotaisles are provided. In the example of FIG. 2, the hot aisles arerepresented as 212, while the cold aisles are represented as 214. Coldair in the plenum 210 is directed through vent tiles 216 in the raisedfloor 208 into the cold aisles 214. The cold air (represented by solidarrows) are directed into the racks 204. Hot air (represented by dashedarrows) exit from the racks 204, and is drawn back to the cooling device206, as indicated by the various dashed arrows in FIG. 2.

The racks 204 can be associated with various temperature sensors,including sensors at the input side and output side of each rack.Temperature sensors at the input side are labeled as 218, while sensorsat the output side are labeled as 220. In other implementations,temperature sensors can be provided at other locations. More generally,the sensors 218 and 220 can provide measurements of any one orcombination of parameters, such as temperature, pressure, relativehumidity, and so forth.

FIG. 2 also depicts a controller 222 according to some implementations.In some examples, the controller 222 can be used to perform the process100 of FIG. 1 for producing models 224 for temperature sensor locations.The controller 222 is also able to use the models 224 for determining arelationship between cooling infrastructure adjustable settings andtemperature sensor locations (or more generally sensor locations). Suchrelationship can be used by the controller 222 for managing the coolingdevice 206 (such as by making adjustments of the adjustable settings ofthe cooling device 206). In the examples where the vent tiles 216 haveadjustable openings, the controller 222 can also control the openings ofthe vent tiles 216.

FIG. 3 is a flow diagram of an analysis process 300 for deriving a TCIand a region of influence, in accordance with some implementations(second stage noted above). The process 300 can be performed by thecontroller 222 of FIG. 2. The process 300 simulates (at 302)perturbation of an adjustable setting of a cooling infrastructure (suchas an adjustable setting of the cooling device 206) using a model (224)that produces predicted temperature sensor data as a function of theperturbed adjustable setting.

The process 300 next computes (at 304) a TCI based on the simulatedperturbation, where the TCI relates a change in temperature at atemperature sensor to a corresponding change in the adjustable setting.To compute the TCI, a local slope of a predicted output from the modelis determined with respect to a particular value of an adjustablesetting. The adjustable setting is varied around the particular value inrelatively small increments—the incremental adjustable setting valuesare provided to the model to produce respective predicted temperaturevalues. A linear function is fitted through the predicted temperaturevalues—this linear function is used to determine the slope, which isused for deriving the TCI.

It is noted that, in accordance with some implementations, TCI is notassumed to be constant throughout different values of an adjustablesettings. In other words, it is possible that the slopes atcorresponding local ranges of adjustable setting values can differ.

Next, using the TCI, the process 300 determines (at 306) a region ofinfluence of a given cooling device, such as the cooling device 206 ofFIG. 2. As discussed further below, a clustering technique can be usedfor determining a region of influence of a given cooling device.

Although FIG. 3 refers to the use of a model and the computation of aTCI, note that the process 300 can also simulate perturbation ofmultiple adjustable settings of the cooling infrastructure usingrespective models. Moreover, multiple TCIs can also be computed forrespective temperature sensors. These TCIs can then be used fordetermining the regions of influence of corresponding cooling devices ofthe cooling infrastructure.

More generally, the analysis process 300 can be applied to derive acorrelation index and a region of influence.

Generating a model (such as performed at 104 in FIG. 1) involvesbuilding the model based on a training data set, which can be part ofhistorical recorded data (such as that received at 102 in FIG. 1). Thehistorical recorded data can be divided into a training data set and atest data set, where the training data set is used for building themodel, while the test data set is used for testing the trained model todetermine an accuracy of the trained model.

In some examples, a model for producing predicted temperature sensordata as a function of at least one adjustable setting can be atree-based model, such as a regression tree. Regression trees can berelatively flexible; for example, they can handle different kinds ofinputs (continuous, discrete, categorical, etc) and perform relativelywell with missing data. In other examples, other types of models can beused.

A regression tree can predict a continuous-valued output, Y (temperaturesensor data), based on at least one input feature, which can berepresented as X₁, X₂, . . . X_(n), where n (n≥1) represents a number offeatures, and X_(i) can represent any of the following adjustablesettings: temperature setting of a cooling device, blower speed, venttile opening setting, or other actuator settings. An “actuator setting”can refer to any adjustable setting of a cooling infrastructure. Theregression tree is able to model a non-linear relationship between theoutput Y and any of the input features X₁, X₂, . . . X_(n).

In some implementations, a regression tree can include a binary tree ofnodes, such as a regression tree 400 shown in FIG. 4. The regressiontree 400 has a root node 402, intermediate nodes 408, 410, and leafnodes 412.

At each root or intermediate node of the regression tree 400, a variabletest can be performed to determine which branch to take from the node.In the example of FIG. 4, the root node 402 performs the following test:a determination of whether the input feature X_(i) is greater than a. Ifso, the right branch 406 is taken; if not, the left branch 404 is taken.

In the regression tree 400, the intermediate node 408 tests whether thefeature X_(j) has the value “high,” and the intermediate node 410 testswhether the feature X_(k) is greater than b. Depending on these tests,corresponding branches are taken until leaf nodes 412 are reached. Eachleaf node corresponds to a respective output value for Y (temperaturesensor data). For example, a first leaf node 412A sets Y=p, and a secondleaf node 412B sets Y=s.

More generally, given an input collection of features (which can be inthe form of an input feature vector), the prediction of the temperaturesensor data using the regression tree 400 involves starting at the rootnode 402 and proceeding through intermediate nodes, and applyingrespective tests at each of the root and intermediate nodes until a leafnode is reached. The input feature vector has corresponding differentvalues for the different features, and such values are tested at therespective root node and intermediate nodes of the regression tree toreach a particular leaf node.

To train a regression tree, the input space (training data set) isrecursively partitioned based on respective features until a relativelysmall number of training data points remain in a partition (thisrelatively small number of training points correspond to a respectiveleaf node and is used to produce a value for Y). In some examples, thevalue of Y at each leaf node can be a constant value that is the average(or other aggregate) of the training data points corresponding to theleaf node.

Training of a regression tree based on a training data set can causeoverfitting of the regression tree in some scenarios. To address issuesrelating to overfitting training data onto a particular regression tree,some implementations can use an ensemble of regression trees (moregenerally an ensemble of models) for each temperature sensor location.Using the models in the ensemble, multiple predicted temperatures of atemperature sensor can be produced based on the corresponding differentmodels of the ensemble. These predicted temperatures can then beaggregated (e.g. by computing an average, a weighted average, a median,a maximum, a minimum, etc.) to produce an output predicted temperatureto be used as the predicted output of the temperature sensor.

In some examples, a random forests technique can be used for producingmultiple models for a given temperature sensor. In other examples, otherensemble learning techniques can be used, such as boosting, bagging,stacking, and so forth.

With the random forests technique, multiple regression trees can becreated from a training data set. With the random forests technique, twomechanisms are provided to reduce correlation between individual modelsin an ensemble. First, to train corresponding regression trees of theensemble, respective different bootstrap samples are selected—use ofdifferent bootstrap samples results in slightly different training datafor each regression tree. Second, to provide randomness, a randomlychosen subset of features can be used at each split in a regression treeduring the training of the regression tree.

Selecting a bootstrap sample from a training data set refers to randomlypicking some number of data points with replacement from the trainingdata set, where a “data point” includes input feature(s) (values of oneor multiple cooling infrastructure adjustable settings) and acorresponding output (temperature sensor data). The bootstrap sample isthen used to fit (train) a regression tree. By selecting just a subsetof data points from the training data set, the remainder of the data canbe considered “out-of-bag data” (or test data) that can be used fortesting of a trained regression tree. For each regression tree, theerror of the regression tree can be computed based on the out-of-bagdata set.

In addition to the above, feature selection can also be used in someimplementations to whittle down the number of features to the mostrelevant ones. Reducing the number of features can result in lesscomplexity, less chances of overfitting, and so forth. Feature selectiontechniques include subset selection, regularization, correlation-basedselection, entropy-based selection, and so forth. After featureselection is performed, the models, e.g. regression trees, can betrained.

FIG. 5 shows an example layout of racks and cooling devices. The coolingdevices are labeled as CRAC1 to CRAC8 in the example. The various racksof electronic devices that are in the layout of FIG. 4 are arranged inmultiple rows of racks, where the rows are labeled A, B, C, D, E, F, G,I, and J. Temperature sensor data can be collected by temperaturesensors mounted at the inlets and outlets of racks depicted in FIG. 5.

FIG. 6 depicts TCI values (along the vertical axis) with respect to thesupply air temperature (SAT) for each cooling device (CRAC1 to CRAC8) asa function of temperature sensors (horizontal axis). For example, apoint 602 represents the TCI value that correlates the temperaturesensor data of a temperature sensor A1 to adjustable setting(s) of thecooling device CRAC1.

To compute a region of influence for each of the cooling devices,clustering of the TCI values can be performed. For example, for TCIvalues associated with the cooling device CRAC1, three clusters 604,606, and 608 can be identified. The clustering generally attempts togenerate dusters of TCI values that are closer to each other than TCIvalues in the other clusters. Examples of clustering techniques that canbe used include any one of the following: K-means clustering, K-medoidsclustering, hierarchical clustering, and so forth.

FIG. 7 is a block diagram of an example arrangement of the controller222 of FIG. 2. The controller 222 can include a model generation module702 for building models according to some implementations. The modelgeneration module 702 can perform the process 100 of FIG. 1, forexample. The controller 222 can also include an analysis module 704 toperform the analysis process 300 of FIG. 3, for example.

The controller 222 can also include a control module 706 for controllingadjustable settings of a cooling infrastructure. The control can bebased on the determined TCI values as well as the determined regions ofinfluence of respective cooling devices. Based on temperaturemeasurements by the various temperature sensors of an arrangement ofelectronic devices, the control module 706 can use the TCI values andregions of influence to adjust adjustable settings of the coolinginfrastructure.

The model generation module 702, analysis module 704, and control module706 can be machine-readable instructions that are executable on one ormultiple processors 708. The processor(s) 708 can be connected to anetwork interface 710 (to allow the controller 222 to communicate over adata network), and a storage medium (or storage media) 712 (to storedata).

Although the various modules 702, 704, and 706 are depicted as beingpart of the same controller 222, note that in alternativeimplementations, the modules can be implemented on separate machines.

The storage medium (or storage media) 712 can be implemented as one ormore computer-readable or machine-readable storage media. The storagemedia include different forms of memory including semiconductor memorydevices such as dynamic or static random access memories (DRAMs orSRAMs), erasable and programmable read-only memories (EPROMs),electrically erasable and programmable read-only memories (EEPROMs) andflash memories; magnetic disks such as fixed, floppy and removabledisks; other magnetic media including tape; optical media such ascompact disks (CDs) or digital video disks (DVDs); or other types ofstorage devices. Note that the instructions discussed above can beprovided on one computer-readable or machine-readable storage medium, oralternatively, can be provided on multiple computer-readable ormachine-readable storage media distributed in a large system havingpossibly plural nodes. Such computer-readable or machine-readablestorage medium or media is (are) considered to be part of an article (orarticle of manufacture). An article or article of manufacture can referto any manufactured single component or multiple components. The storagemedium or media can be located either in the machine running themachine-readable instructions, or located at a remote site from whichmachine-readable instructions can be downloaded over a network forexecution.

In the foregoing description, numerous details are set forth to providean understanding of the subject disclosed herein. However,implementations may be practiced without some or all of these details.Other implementations may include modifications and variations from thedetails discussed above. It is intended that the appended claims coversuch modifications and variations.

What is claimed is:
 1. A method executed by a system having a processor,comprising: receiving recorded data including sensor data and anadjustable setting of a cooling infrastructure; generating a model ofimpact of the cooling infrastructure based on the recorded data, wherethe model produces predicted sensor data as a function of at least oneinput feature that includes the adjustable setting, and where the modelis able to model a non-linear relationship between the predicted sensordata and the adjustable setting, wherein the model is for a particularsensor; generating another model based on the recorded data, where theanother model produces predicted sensor data output by the particularsensor as a function of the at least one input feature, wherein themodels are part of an ensemble of models for the particular sensor;using the models in the ensemble to produce respective predictedmeasurements from the particular sensor; and aggregating the predictedmeasurements to produce an output predicted sensor data for theparticular sensor.
 2. The method of claim 1, wherein the adjustablesetting is selected from among a temperature setting of a cooling devicein the cooling infrastructure, a blower speed of a cooling device in thecooling infrastructure, and a setting of an adjustable openingassociated with an airflow conduit in the cooling infrastructure.
 3. Themethod of claim 1, wherein the sensor data is selected from amongtemperature sensor data, pressure sensor data, and relative humiditysensor data.
 4. The method of claim 1, further comprising: computing acorrelation index using the model, where the correlation index relates achange in a parameter at a sensor to a corresponding change in theadjustable setting.
 5. The method of claim 4, wherein computing thecorrelation index comprises computing a thermal correlation index. 6.The method of claim 5, further comprising: determining a region ofinfluence of a cooling device in the cooling infrastructure based on thethermal correlation index.
 7. The method of claim 1, further comprisingadjusting, by the system based on the predicted sensor data for theparticular sensor, the adjustable setting of the cooling infrastructure.8. The method of claim 1, wherein the adjustable setting of the coolinginfrastructure is an adjustable setting of an adjustable opening of afloor vent tile through which cooling air passes from a plenumunderneath a floor to equipment above the floor.
 9. The method of claim1, wherein using the models in the ensemble to produce the respectivepredicted measurements comprises using the models in the ensemble toproduce respective predicted temperatures from the particular sensor,and wherein aggregating the predicted measurements to produce the outputpredicted sensor data for the particular sensor comprises aggregatingthe predicted temperatures.
 10. The method of claim 1, wherein theaggregating comprises computing one of an average, a median, a maximum,or a minimum of the predicted measurements to produce the outputpredicted sensor data for the particular sensor.
 11. An articlecomprising at least one non-transitory machine-readable storage mediumstoring instructions that upon execution cause a system to: simulateperturbation of an adjustable setting of a cooling infrastructure usinga plurality of different models that produce respective predictedtemperatures for an individual sensor as a function of the adjustablesetting, the simulating aggregating the respective predictedtemperatures output by the plurality of different models to produce anaggregate predicted temperature for the individual sensor; and determinea relationship between the adjustable setting and temperatures of theindividual sensor based on the simulated perturbation; and controladjustment of the adjustable setting of the cooling infrastructure basedon the determined relationship.
 12. The article of claim 11, whereindetermining the relationship includes computing a thermal correlationindex that relates a change in a temperature at the individual sensor toa corresponding change in the adjustable setting.
 13. The article ofclaim 11, wherein the simulating comprises simulating perturbations ofadjustable settings of cooling devices in the cooling infrastructure,and wherein the instructions upon execution cause the system to computecorrelation indexes for corresponding plural sensors based on theperturbations.
 14. The article of claim 11, wherein the plurality ofdifferent models comprise regression trees, and wherein the instructionsupon execution cause the system to further train the regression treesusing recorded data including measurements at a plurality of sensors,and corresponding adjustable settings of cooling devices in the coolinginfrastructure.
 15. The article of claim 11, wherein the aggregatingcomprises computing one of an average, a median, a maximum, or a minimumof the predicted temperatures to produce the aggregate predictedtemperature for the individual sensor.
 16. A system comprising: at leastone processor to: receive recorded data including measurement data of asensor, and an adjustable setting of a cooling infrastructure; generatea plurality of different models for the sensor based on the recordeddata, where each of the plurality of different models produces arespective predicted sensor data as a function of an input feature thatincludes the adjustable setting, and where at least a given one of theplurality of different models is able to model a non-linear relationshipbetween a respective predicted sensor data and the adjustable setting;aggregate the respective predicted sensor data produced by the pluralityof different models to compute an output predicted sensor data for thesensor, the aggregating comprising computing one of an average, amedian, a maximum, or a minimum of the respective predicted sensor dataproduced by the plurality of different models; and control adjustment ofthe adjustable setting of the cooling infrastructure based on the outputpredicted sensor data for the sensor.
 17. The system of claim 16,wherein the at least one processor is to further: generate a thermalcorrelation index for the sensor using the output predicted sensordevice; and determine a region of influence of a cooling device of thecooling infrastructure using the thermal correlation index.
 18. Thesystem of claim 16, wherein the respective predicted sensor dataproduced by the plurality of different models comprise respectivepredicted temperatures for the sensor, and wherein the aggregating ofthe respective predicted sensor data produced by the plurality ofdifferent models to compute the output predicted sensor data for thesensor comprises aggregating the respective predicted temperatures.